The capture and representation of 3D geometric surface images is central to a number of image processing tasks, from animation to moviemaking to medical imaging. The traditional approaches to representing images of objects is based on the ability of the human eye and cameras to perceive amplitude of light. In one form or another, an amplitude map of light received at a camera serves as a representation of a captured image, which can then be processed, stored, transmitted, etc.
Likewise, computer-generated imagery for use in special effects and other computer graphics applications, such as animation, typically is based on amplitude representations of an image. These may be monochrome or color amplitude mappings. More recently, optical phase shifting techniques have been shown to advantageously provide high measurement resolution. Phase shifting techniques include generating phase-shifted optical fringe patterns, projecting the patterns onto a surface of an object, capturing images of reflections of the incident patterns as distorted by the surface of the object, and processing the reflected images to extract the phase of each of their pixels while removing phase discontinuities (a process called “phase unwrapping”). Following the phase unwrapping, absolute coordinates of the surface of the object are calculated from the derived phase information. The term “absolute coordinates” refers to the fact that the coordinates are based on a particular point of reference, which can be chosen arbitrarily. U.S. Pat. No. 6,788,210, incorporated herein by reference, discloses recent achievements in 3D surface contouring based on using phase shifting techniques.
Applications that depend on 3D surface measurement increasingly often require capturing the 3D geometry of objects very fast, to permit capture of dynamic changes of the surfaces of the objects, or to permit responsive action. As a case in point, in some areas of the entertainment industry, such as, for example, in computer animation and video games, an ability to capture and reproduce the 3D geometry of a face and/or body in real-time is desired. However, currently, 3D geometric motion capture requires attaching sensors to a subject (an actor, for example). This process is cumbersome and time-consuming, and the presence and size of the sensors limit the amount of detail that can be captured. Similar problems exist in medical imaging, wherein not entirely portable and very slow three-dimensional scanners are employed. Usually, transporting the patient to the scanner is required. This is often a difficult experience for the patient who must sit completely motionless for a relatively long time, which, depending on the patient's condition, may be uncomfortable, painful, and even unrealistic. Capturing the 3D coordinates of the moving object without resorting to bulky sensors and in real-time would largely alleviate these serious shortcomings—at least for certain kinds of imaging needs (e.g., surface, not internal imaging). Real-time 3D surface measurement can potentially revolutionize some aspects of medical diagnostics and treatment. By analyzing real-time 3D measurements of moving human muscles, it may be possible to determine, for example, whether the muscles are affected by a certain disease or whether a nerve, rather then the muscles, causes the problem. By analyzing the motion of the chest, a pulmonary problem might be diagnosed. By analyzing the motion of the breast, a noninvasive diagnosis of certain conditions is conceivably possible. Real-time 3D scanning can also improve design of existing medical devices, such as X-ray or magnetic resonance imaging (MRI) machines, by providing feedback for determining whether the patient is positioned correctly during the scanning and imaging. Real-time 3D scanning can also be utilized in many homeland security applications as well as in scientific research and manufacturing (e.g., for quality control).
Phase shifting techniques applied to optical imaging depend (in the context of the invention), as the name implies, on generating and detecting phase shifts in optical signals. Historically, phase shifting of optical signals was achieved with mechanical gratings. More recently, a phase shifting technique has been developed wherein digitally-interpretable fringe pattern signals are generated electrically, converted electro-mechanically into optical phase-shifted fringe patterns (or images) and emitted via a projector.
Traditionally, in such a contour measuring optical system, at least three phase-shifted fringe patterns are shone on an object, with their reflections, together with an image of the object (i.e., a conventional amplitude image), being captured by a digital camera whose output is processed to obtain the 3D geometry data corresponding to the surface of the object. Increasing the number of phase-shifted images results in higher accuracy in the 3D geometry measurement. However, adding fringe patterns requires additional processing, which necessitates an increase in resources and time. The fewer the number of fringe images, the faster the speed that can be achieved. Real-time contour measurement is quite expensive with three or more phase-shifted images to be processed, and may not be realistic with commercially available projectors and cameras, and the processing required for contours to appear smooth to the human eye. (Reconstruction of a contour of an object is currently considered to be performed in real-time if the image acquisition speed is at least 30 frames per second, where a frame is a unique image.)
Furthermore, in many applications, it is desirable to be able to capture surface texture simultaneously with acquiring the 3D geometry. To obtain texture of acceptable quality when three phase-shifted fringe images are used, it is required that fringes have very precise phase shift and that the profiles of the fringes be ideally sinusoidal; otherwise, the quality of the captured texture may be compromised.
The speed of acquisition of the 3D geometry depends also on characteristics of the phase wrapping and unwrapping algorithms that are employed. The phase obtained from fringe images normally ranges from 0 to 2π radians. When multiple fringes are used, phase discontinuities occur every time the phase changes by 2π. Phase unwrapping is necessary to remove the 2π ambiguities on the wrapped phase image, thus obtaining a continuous phase map. A key to successful phase unwrapping is an ability to accurately detect and correct for the 2π discontinuities. For images with noise and images of complex geometric surfaces and surfaces with sharp edges, phase unwrapping is usually difficult. Various algorithms have been developed for phase unwrapping; however, they are generally too slow for capturing the 3D geometry in real-time and with reasonable cost.
Conventional phase unwrapping results in a relative phase map, or a relative phase unwrapped image, i.e. it does not refer to a point of reference. To obtain an absolute phase map, an absolute phase mark signal may be generated and projected onto the surface of the object, as described in U.S. Pat. No. 6,788,210. The absolute phase map is obtained from the relative phase map by subtracting the phase of the absolute phase mark from every pixel of the relative phase map. While this method is efficient, particularly when the absolute phase mark is a line, it requires taking an additional image of the mark on the object, which increases the processing requirements and correspondingly decreases the speed of the 3D geometry acquisition.
After the absolute phase has been obtained and a calibration has been carried out to establish relationships between a camera, projector, and a reference whose coordinates are known, absolute coordinates of the surface of the object can be determined. The calculations of the absolute coordinates are computationally intensive, and the process can therefore be a limiting step in performing real-time 3D surface measurement.
Conventionally, the captured absolute phase and amplitude information is then transformed into a mesh of triangles which provide piece-wise linear representation of the surface of the object but loose information that was present in the phase. Such a mesh is a common framework for the representation of surfaces in the field of computer graphics and in geometric as well as surface property modeling. Most graphics processors “expect” surfaces to be represented in this fashion. Likewise, computer-generated fabrications of images (e.g., in animation software) typically represent objects as meshes of small triangles representing spatial positions with only amplitude (e.g., color) information as textures. While mesh representations are convenient, they lack the fill information content of the phase information and require considerable memory.
In view of the aforesaid, there is a need for an accurate and fast absolute coordinate 3D surface imaging method and system and a better image representation format. Accordingly, the present invention was conceived at least partly with a goal of eliminating the above-described disadvantages of existing methods and systems, and example embodiments are directed to providing methods and apparatus to capture 3D geometry quite fast—preferably, in real-time, and to more efficiently and robustly representing 3D geometric surfaces.